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Abstract. We use language theory to study the rational subset prob- 
lem for groups and monoids. We show that the decidability of this 
problem is preserved under graph of groups constructions with finite 
edge groups. In particular, it passes through free products amalga- 
mated over finite subgroups and HNN extensions with finite associated 
subgroups. We provide a simple proof of a result of Grunschlag showing 
that the decidability of this problem is a virtual property. We prove 
further that the problem is decidable for a direct product of a group 
G with a monoid M if and only if membership is uniformly decidable 
for G-automata subsets of M. It follows that a direct product of a free 
group with any abelian group or commutative monoid has decidable 
rational subset membership. 



1. Introduction 

A key aspect of combinatorial group theory is the study of algorithmic 
decision problems in finitely generated groups. This trend, which was ini- 
tiated by Dehn jJUj, has always remained central to the subject, and has 
recently been given new impetus by interest in the potential use of group 
theory as a basis for the development of secure cryptographic systems (see, 
for example, inZHSHl)- The best known example of a group-theoretic decision 
problem is the word problem for a given finitely generated group G: given 
two words over the generators, decide if they represent the same element, 
or equivalently, given a word over the generators, decide if it represents the 
identity. 

A natural generalization is the subgroup membership problem or gener- 
alized word problem: given elements and g oi G (specified as 
words over the generators), decide if g lies in the subgroup generated by 
ui,...,Un (Sni- More generally still, one can consider the corresponding 
problem for finitely generated submonoids and subsemigroups of groups; 
Margolis, Meakin and Sunik [^31 have recently solved the submonoid mem- 
bership problem for a large class of groups by studying distortion functions. 
Decidability of the subsemigroup problem is of particular interest since it 
also implies solvability of the order problem. 

From the point of view of computer science and formal language theory, 
finitely generated subgroups, submonoids and subsemigroups are examples 
of rational subsets and it is natural also to consider the harder rational 
subset problem for G: given a rational subset of G (specified using a finite 
automaton over the generators) and a word representing an element of G, 
decide if the latter belongs to the former. 

It is well-known that the subgroup membership problem, and more gen- 
erally the rational subset problem, is decidable for free groups and for free 
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abelian groups |Hl|^|Snj. Many important groups can be built up from 
these groups using constructions such as direct products, free products (with 
and without amalgamation), HNN extensions and graphs of groups |3U1 138j . 
Hence, a natural next step is to consider the extent to which decidability of 
these algorithmic problems is preserved under such operations. 

Kapovich, Weidmann and Myasnikov |2Zj have recently studied the sub- 
group membership problem in fundamental groups of graphs of groups. It 
is natural to ask whether their results can be extended to the more general 
rational subset problem. Their approach generalizes the well-known folding 
technique of Stallings [lOj , which starts with a graph easily constructed from 
the subgroup generators, and incrementally computes the important part of 
the Schreier graph associated to a finitely generated subgroup of the free 
group. This method is essentially automata-theoretic, since the graphs can 
be viewed as automata recognising progressively larger parts of the mem- 
bership language of the subgroup |2llS2||^. However, the folding technique 
relies upon a degree of symmetry in the automata (the dual automaton prop- 
erty, in the language of |41J which is particular to the subgroup case, and 
so does not easily generalize. 

The main aim of the present paper is to make a start upon the study of 
the relationship between the rational subset problem and the constructions 
described above. Our approach is largely language-theoretic; in Section |21 
we introduce an abstract property of languages which corresponds naturally 
to decidability of rational subset membership. This simple observation es- 
tablishes a surprisingly deep connection between algorithmic group theory 
and formal language theory. It immediately yields simple language-theoretic 
proofs of several non-trivial known results, including the fact that decidabil- 
ity of the rational subset problem is a virtual property, which was first proved 
by Grunschlag |19J; recall that a virtual property of a group is a property 
inherited from finite index subgroups and overgroups. Conversely, we are 
also able to use undecidability of a group-theoretic problem to establish a 
new undecidability result in language theory: there exists a context-free 
language for which it is undecidable which regular languages it contains. 

In Section 2] we proceed to show that our abstract property is preserved 
when taking the ancestor set of a language under certain infinite rewriting 
systems. In Section we apply these results to show that decidability of 
rational subset membership is preserved under graph of groups constructions 
j3Sj with finite edge groups. It follows in particular that it passes through 
amalgamated free products over finite subgroups, and HNN extensions with 
finite associated subgroups. 

Section El introduces a connection between decidability of rational subsets 
and the theory of G-automata (see for example ^1 E^), that is, rational 
transductions of group word problems. Specifically, we show that a direct 
product of the form GxM has decidable rational subset membership exactly 
if membership is (uniformly) decidable for G-automata subsets of M. This 
combines with a group-theoretic interpretation |2H] of a theorem of Chomsky 
and Schiitzenberger [Hj and some classical results on commutative monoids 
jl4l I36j to show that any direct product of a free group with an abelian 
group (or commutative monoid) has decidable rational subset membership. 
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Finally, in Section [7| we consider the subgroup membership problem and 
rational subset problem in the important class of graph groups (which are 
also known as right-angled Artin groups, trace groups or free partially com- 
mutative groups). We note some consequences for certain of these groups of 
our results from Sections El and El and pose a number of questions regarding 
other graph groups. 

While the primary focus of this paper is on groups, the rational subset 
problem and associated decision problems are also of interest in more general 
monoids and semigroups. For example, recent research of Ivanov, Margolis 
and Meakin [22] has reduced the word problem for a large class of one-relator 
inverse monoids to the submonoid problem in one-relator groups. Since de- 
cidability of rational subset membership is inherited by finitely generated 
subsemigroups, many of our results about groups have additional implica- 
tions for monoids and semigroups. We also prove directly some new results 
for monoids, including the fact that the rational subset problem is decidable 
for direct products of free groups with free monoids. 

2. Preliminaries 

In this section, we provide a brief introduction to rational subsets and 
associated decision problems. 

2.1. Rational Subsets, Languages and Transductions. Let M be a 

finitely generated monoid. Recall that a finite automaton P over M is a finite 
directed graph with edges labelled by elements of M, with a distinguished 
initial vertex and a set of distinguished terminal vertices. The labelling 
of edges extends naturally, via the multiplication in M, to a labelling of 
paths by elements of M. The subset recognised by the automaton is the set 
of all elements of M which label paths between the initial vertex and some 
terminal vertex. A subset of M recognised by some finite automaton is called 
a rational subset of M. An alternative description of the rational subsets of 
M is as the smallest collection of subsets of M containing the finite subsets 
and closed under union, concatenation and generation of submonoids. 

A particularly important case arises when M = S* is the free monoid on 
an alphabet S, so that the automaton P accepts a language over S, which we 
denote L[P). A rational subset of a free monoid is called a rational language 
or regular language. By Proposition III.2.2], the rational subsets of a 
monoid M are exactly the homomorphic images of regular languages. For a 
detailed introduction to the theory of regular languages, including a number 
of alternative definitions, see |13j or |22|. 

Another significant case is when M = E* x O* is a direct product of free 
monoids. A finite automaton over S* x Q* is called a finite transducer from 
S* to 0*. A subset recognised by a finite transducer, that is, a rational 
subset of E* X il*, is called a rational transduction. If p C S* x $7* and 
L C S* then the image of L under p is the language 

L/9 = {w € 0* I (u, v) G p for some n € L} C Cl*. 

If /9 is a rational transduction then we say that the language Lp is a rational 
transduction of the language L. Rational transductions are a powerful tool 
for studying relationships between formal languages, and many elementary 
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operations on languages are instances of rational transductions. For exam- 
ple, given a word G S* it is straightforward to show that there is an 
effectively constructible rational transduction taking each language L C S* 
to its left translation by w, that is, the language wL = {wx | x G L}; an 
analogous result holds for right translation. More general information about 
rational transductions can be found in [H]. 

2.2. Encodings and Decision Problems. We shall work with algorithmic 
problems in which abstract objects such as monoid elements, automata and 
languages are regarded as data. Formally, it is necessary to have an agreed 
system of (not necessarily unique) finite encodings for such objects, but for 
brevity and clarity it is unhelpful to refer repeatedly to encodings. Typically 
we shall introduce an encoding along with the definition of a class of objects, 
but thereafter leave the encoding implicit. 

All groups and monoids in this paper will be finitely generated, and we 
assume that elements are encoded as words over some (except where oth- 
erwise stated, fixed) finite monoid generating set. Our algorithms are not 
uniform across groups and monoids, and so we do not need to consider an 
encoding system for groups and monoids themselves. 

Finite automata are presumed to have some suitable encoding, from which 
we can extract the vertex set and for any pairs of vertices p and q, the 
(encodings of) labels of all edges from p to q. Languages and subsets are 
typically of a type recognised by some kind of algorithm or automaton or 
generated by some kind of grammar, and we assume they are encoded as an 
automaton, algorithm or grammar. 

Now let C be a set of subsets of a finitely generated monoid M, and 
suppose we have a fixed system of (not necessarily unique) encodings for 
elements of C. For example, C might be the set of rational subsets of M, 
encoded as finite automata. We say that membership is uniformly decidable 
for C if there is an algorithm which, given an (encoded) set 5" € C, and an 
(encoded) element m € M, decides if m lies in S. The following elementary 
proposition says that this property is independent of the choice of finite 
generating set for M. 

Proposition 2.1. Let X and Y be finite generating sets for a monoid M, 
and let C be a set of subsets of M with a system of encoding for its elements. 
Suppose there exists an algorithm which, given an (encoded) set S G C and 
a word u G X* , decides if the word u represents an element of the set S. 
Then there exists an algorithm which, given an (encoded) set S (z C and a 
word V gY* , decides if the word v represents an element of the set S. 

Proof. For each letter y gY, let Wy G X* be a word representing the same 
element of M as y. Define a morphism p : Y* ^ X* hy y ^ Wy for all 
y GY . Now a word v gY* represents the same element of M as vp G X* . 
Hence, it suffices to take the algorithm which, given as input an encoded set 
S £ C and a word v £ Y*, computes the word vp G X* by replacing each 
letter y of v with Wy, and then uses the algorithm given to check whether 
vp represents an element of S. □ 

One instance of this problem forms the main focus of this paper. A monoid 
or group M has decidable rational subset problem if membership is uniformly 
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decidable for the rational subsets of M (encoded as finite automata over the 
generating set). Proposition 12.11 savs that this property is invariant under 
changing the generating set used to specify the element; much the same 
argument shows that it is also invariant under change of the generator set 
used for the edge labels in the automaton. Similarly, if a finitely generated 
monoid M has decidable rational subset problem, then it is clear that if A'^ 
is a finitely generated submonoid of M then the rational subset problem is 
also decidable for A^. Moreover, given an algorithm for the rational subset 
problem in M and a finite set of generators for the submonoid N, we can 
effectively compute an algorithm for the rational subset problem in A^. 

Decidability of the rational subset problem for a group G implies decid- 
ability of the word problem and the subgroup, subsemigroup and submonoid 
membership problems. Since an element g € G has finite order exactly if 
the subsemigroup generated by g contains the identity, an algorithm for the 
rational subset problem also allows one to decide if a given element has finite 
order. Moreover, if g does have finite order then, since the word problem is 
solvable, it is a simple matter to compute the order by enumerating words 
representing powers and checking if they represent the identity. 

2.3. Groups and Word Problems. If G is a finitely generated group 
generated by a subset A then the word problem Wa{G) for G with respect 
to A is the set of all words in A* which represent the identity in G. More 
generally, given a monoid M generated by a subset A and an element m € M, 
we denote by Wa{M, m) the set of all words in A* representing the element 
m G M. 

3. Regular Intersection Decidability 

In this section we observe that decidability of the rational subset problem 
in a group is equivalent to a natural language-theoretic property of the word 
problem. By studying the abstract class of languages with this property, 
we show that many existing results about the rational subset problem for 
groups can be easily deduced from standard results in language theory. In 
Section |3 we shall see that this property of languages is preserved when 
taking ancestor sets under certain infinite rewriting systems. In Sectional 
we shall apply these results to the rational subset problem for fundamental 
groups of graphs of groups, and hence for amalgamated free products and 
HNN extensions. 

We begin by introducing an algorithmic problem that can be associated 
to any formal language. Let L C S* be a language. The regular intersection 
decision problem or RID problem for L is the problem of deciding, given a 
finite automaton over S, whether the language recognised intersects (that is, 
has non-empty intersection) with L. An algorithm which solves this problem 
is called an RID algorithm for L, and if there exists such an algorithm then 
L is called regular intersection decidable or RID. It is easily seen that an 
RID algorithm solves the RID problem for a unique language; hence, RID 
languages can be encoded (although not uniquely) as algorithms which solve 
their RID problem. 

The following provides the connection with the rational subset problem 
for groups, and is our motivation for studying RID languages. 
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Theorem 3.1. Let G be a group generated by a finite subset A. Then the 
following are (effectively) equivalent: 

(i) rational subset membership is decidable for G; 

(ii) the word problem for G with respect to A is RID; 

(iii) the RID problem for Wa{G, g) is decidable uniformly in g ^ G (where 
elements of G are encoded as words in A* ). 

Proof. Let i? be a rational subset of G encoded as a regular language L of 
A* . Then given an element g € G, one has 5 G i? if and only if L intersects 
with WAiG,g). This establishes the equivalence of (i) and (iii). 

That (iii) implies (ii) is immediate. For the converse, let w (z A* represent 
g (z G and let L C A* be a regular language. Then, since (ii) clearly allows 
us to solve the word problem, we can effectively compute a word u (z A* 
representing g~^. Moreover, we can effectively compute the regular language 
uL. Now L intersects with WAiG,g) if and only if Wa{G) intersects with 
uL; by assumption, we can test the latter. □ 

Despite the simplicity of the proof, we shall see that Theorem 13. II estab- 
lishes a surprisingly deep connection between algorithmic group theory and 
formal language theory. We observe that the equivalence of (i) and (iii) holds 
also for monoids. The following proposition summarises some elementary 
properties of the class of RID languages. 

Proposition 3.2. The class of RID languages is effectively closed under 
union and rational transduction (and hence also under morphism, inverse 
morphism, intersection with regular languages and right and left transla- 
tion). The class of RID languages is strictly contained within the class of 
recursive languages, and contains the class of indexed languages ( and hence 
also the context-free and regular languages) . 

Proof. If L and K are RID languages and i? is a regular language then R 
intersects with LU K exactly if it intersects with L or so closure under 
union is clear. 

For closure under rational transduction, suppose L C A* is RID and 
(7 C j4* X i?* is a rational transduction; we must show that the language 

K = La = {b £ B* I (a, 6) € o" for some a € L} 

is RID. To this end, suppose we are given a regular language Q C B*. Let 

P = Qa'^ = {a e A* \ (a, b) e a for some b G Q}. 

Then P is a regular language, and moreover can be effectively computed 
from Q (SI Corollary III. 4. 2]. Now it is easily verified that K intersects 
with Q if and only if L intersects with P; by assumption, the latter can be 
checked. 

The class of indexed languages (encoded as indexed grammars) is effec- 
tively closed under intersection with regular languages ^ Corollary 3], and 
has decidable emptiness problem ^ Theorem 4.1]. Thus, one can test if a 
regular language R intersects with an indexed language L by computing the 
intersection L f] R, and testing it for emptiness. Hence, indexed languages 
are RID. Since regular and context-free languages are indexed il|, it follows 
also that these languages are RID. 
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Since singleton sets are (effectively computable as) regular sets, the mem- 
bership problem for a language is reducible to the RID problem, so RID 
languages are recursive. On the other hand, the group F2 x F2 has solvable 
word problem but undecidable rational subset problem j3UL I34j . Hence, by 
Theorem 13. 11 its word problem is recursive but not RID. □ 

We now discuss a number of consequences of Theorem 13.11 and Proposi- 
tion The following lemma will combine with these results to show that 
the decidability of the rational subset problem is inherited by finite index 
overgroups, and hence is a virtual property. The idea, which has been used 
by several authors ^lElHO]) is essentially a recoding of the Kaloujnine- 
Krasner embedding For completeness, we state and prove the lemma 
explicitly. 

Lemma 3.3. Let G he a finitely generated group and H a finite index sub- 
group. Let X be a finite generating set for G and Y he a finite generating 
set for H . Then there is a rational transduction cr C y* x X* such that 
Wx{G) = WYiH)a. 

Proof. Let gi, . . . , gn he a complete set of right coset representatives of H in 
G, assuming without loss of generality that gi is the identity. For each x S X 
and i S {1, . . . , n}, choose a word Wi^x £ Y* representing the unique element 
hi^x S H such that giX = hi^xdj, where HgiX = Hgj. Our transducer has 

vertex set G/H. The labelled edges are of the form Hgi HgiX with 
X G X, i G {!,... ,n}. The initial vertex and terminal vertex are both 
the coset H. If a C y* x X* is the associated rational transduction, then 
it is easy to see that a^^ : X* ^ y* is a partial function such that, for 
w G X*, wa~^ is defined if and only if w represents an element of H, in 
which case wa"^ is an element of Y* representing the same element of H as 
w; in particular w G Wx{G) if and only if wa~^ G Wy{H). It follows that 
WYiH)a = WxiG). □ 

Theorem 13.11 Proposition 13.21 and Lemma 13.31 immediatelv yield the fol- 
lowing result of Grunschlag 19 . 

Corollary 3.4. Let G and H be finitely generated groups such that H is a 
finite index subgroup of G and suppose that H has decidable rational subset 
membership. Then G has decidable rational subset membership. 

The easier part of a celebrated theorem of Muller and Schupp [HS] states 
that every finitely generated virtually free group has context-free word prob- 
lem. Combining this with Theorem 13 . 1 1 and Proposition 13. 21 we immediately 
obtain the following result which subsumes a result of Benois [Hj. A proof 
of this nature was first suggested by Margolis and Meakin |31j . 

Corollary 3.5. Finitely generated virtually free groups have decidable ra- 
tional subset problem. 

A significant open question is that of which finitely generated groups 
have word problems which are indexed languages ^ |21 1181 129j ; an answer is 
likely to be of significant interest in both group theory and language theory. 
Theorem 13.11 and Proposition 13.21 give an alternative proof of the following 
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result of Lisovik j^H] , which gives a necessary condition for a group to have 
indexed word problem. 

Corollary 3.6. Let G be a finitely generated group with indexed word prob- 
lem. Then G has decidable rational subset membership (and hence decidable 
subgroup membership problem and order problem). 

We can also use Theorem to obtain some purely language theoretic 
results. We have already observed that the group F2 x F2 has undecidable 
subgroup membership problem ,30yj34] and hence, by Theorem 13.11 non-RID 
word problem. In j2()j . it is shown that the word problem of this group is 
the complement of a context-free language. Combining these two we obtain 
the following. 

Corollary 3.7. The class of complements of context-free languages is not 
contained in the class of RID languages. 

Since a regular language intersects with a language L exactly if it is not 
contained in the complement of L, Corollarv l8.7l is equivalent to the following 
statement. 

Corollary 3.8. There exists a context-free language L such that there is no 
algorithm which decides, given a regular language R, whether R is contained 
in L. 

We note that the related problems in which the regular language is fixed 
and the context-free language varies are also undecidable in general, as a 
consequence of undecidability of completeness for context-free languages; a 
detailed study of these problems can be found in fH] . 

4. RID Languages and Monadic Rewriting Systems 

In this section, we show that the class of RID languages is closed under the 
operation of taking ancestor sets with respect to certain infinite rewriting 
systems. In Section El we shall use this result to show that decidability of 
the rational subset problem passes through graph of groups constructions 
with finite edge groups. 

A monadic rewriting system T over a finite alphabet S is a subset of 
S* X (S U {e}). An element {w, x) € F is normally written w ^ x. If C is a 
class of languages, then T is called a C-monadic rewriting system if for each 
X G S U {e}, the set 

r,^ = {w e^* \{w ^ x) £ r} 

belongs to C. A finite encoding system for the class C naturally gives rise to 
a finite encoding system for C-monadic rewriting systems which stores for 
each X E S U {e} an encoding of F^. We shall be particularly interested in 
the case where C is the class of RID languages, encoded via RID algorithms. 

If F is a monadic rewriting system, then we write u ^ v ii u = rws G E* 
and V = rxs G S* with w — > x G F. We denote by the reflexive, transitive 
closure of the relation If u v, then v is said to be a descendant of 
u (under F) and u is said be an ancestor of v (under F). If L C S* is a 
language, then LT denotes the set of all descendants of L under F and LF~^ 
denotes the set of all ancestors of L under F. 
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It is well-known to computer scientists that the set of descendants of 
a regular language under a monadic rewriting system is again a regular 
language. Moreover, if the rewriting system is finite |1J |H] or context-free 
[7] then one can algorithmically construct an automaton for the language of 
descendants. The following theorem gives a more general condition under 
which this is possible. 

Theorem 4.1. Let F C S* x (S U {e}) be a monadic rewriting system and 
let L CT,* be a regular language. Then LT is regular. Moreover, there is an 
algorithm which, given an RID monadic rewriting system (encoded via the 
RID algorithms for the F^., x G S U {e} ) and a finite automaton recognising 
a language L C S*, produces an automaton recognising LT. 

Proof. Let Mq be a finite automaton recognising a language L C S*. Clearly, 
by subdividing edges and adding extra vertices as necessary, we may assume 
without loss of generality that the edges of Mq are labelled by elements of 
S U {e}. Now starting from Mq, we construct a sequence of automata as 
follows. 

The automaton Mj+i has the same vertex set as Mj, and all the edges 
of Mi (which we call inherited edges in Mj+i), plus some additional edges 
(called new edges) constructed as follows. For each pair of vertices p and q 
in Mi denote by L^^ the set of all words labelling paths from p to g in Mj. 
For each element x € S U {e} such that F^; n L^^ 7^ 0, Mj+i is given a new 
edge from p to q labelled x (unless M, already had one). 

Moreover, Lp^ is easily seen to be (effectively computable as) a regular 
language. Hence, if we are given F encoded using RID algorithms, we can 
test whether each F^, intersects with L^^ and so Mj+i can be effectively 
constructed from Mj. 

Since every automaton in the sequence has the same vertex set, and at 
each stage we only add edges labelled by the (finitely many) letters in Suje} 
in places where they do not already exist, the sequence must terminate. 
That is, there exists j such that = Mj for all k > j. We claim that 
LT = L{Mk). 

Clearly L{Mi) C L(Mj_|_i) for each i since we have been adding new edges. 
We claim that if f G L(Mi) and v ^ w, then vu € -L(Mj_|_i). Indeed, suppose 
V = rus and w = rxs with u — > x G F. Then v labels a successful path 
TT in Mj and there is a factorisation it = pTa such that the paths p, T, a 
are respectively labelled by r, u, s. Let e,/ be the respective initial and 
terminal vertices of the path T. Then either an edge ^ from e to / labelled 
by X already exists in Mj, or a new edge ^ from e to f with label x is added 
in the construction of Mj+i. Hence p^o" is a successful path labelled by 
w = rxs in Mj+i. It follows immediately that if u G L and u =^>* w, then 
w G L{Mr) where r is the number of steps needed to derive w from u. Hence 
LT C L(Mfc). 

To show the converse, it suffices to show that -L(Mj+i) C L(Mj)F. Sup- 
pose w G -L(Mj_|_i). Then Mj+i has a path vr from the initial vertex to some 
terminal vertex labelled w; consider a factorisation w = wqXiWiX2W2 . . . XnW„ 
where, in the path vr, each Wj G S* is read along inherited edges and each 
Xj G S U {e} is read along a new edge from ej to fj. Now by the con- 
struction of Mj, for each xj there exists yj G S* such that yj Xj £ T 
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and Uj labels a path from ej to fj in Mj. If follows that the word v = 
woUiWi ■ . - UnWu & L{Mi). Since v w, we see that L(Mj+i) C L{Mi)T. 
Thus L{Mk) C L(Mo)r = LT, as required. □ 

In it is shown that the set of ancestors of a context-free language 
under a context-free monadic rewriting system is always context-free. We 
obtain an analogous effective result for RID rewriting systems. 

Corollary 4.2. Let L T,* be an RID language and let T be an RID 

monadic rewriting system over S. Then LF"^ is RID. Moreover, there is 
an algorithm which, given an RID language (encoded as an RID algorithm) 
and an RID monadic rewriting system T ( encoded as above ), outputs an RID 
algorithm for LT^^ . 

Proof. Let R Q T,* . Then it is straightforward to verify that R intersects 
with LT~^ if and only if RT intersects with L. 

The corollary is now immediate from Theorem 14. II since a finite automa- 
ton recognising RT can be effectively constructed and we can use the RID 
algorithm for L to check whether RT intersects with L. □ 

This corollary allows a new interpretation of the rational subset problem. 

Corollary 4.3. Let G be a group with finite generating set A. Then the 
following are equivalent: 

(i) G has a decidable rational subset problem; 

(ii) Wa{G) is RID; 

(iii) Wa{G) = {e}r^^ for some RID monadic rewriting system T. 

Proof. Theorem gives the equivalence of (i) and (ii). To show that (ii) 
implies (iii) simply take the RID monadic rewriting system consisting of all 
rules w ^ e where w belongs to the word problem. The implication (iii) 
implies (ii) is an immediate consequence of Corollary 14.21 and the fact that 
singletons are RID languages. □ 

5. Graphs of Groups, Amalgamated Products and HNN 

Extensions 

In this section, we apply the language-theoretic results of Section |31 to 
some problems in group theory. We show that decidability of rational subset 
membership is preserved under graph of groups constructions jHHj with finite 
edge groups. A particular consequence is that this property passes through 
free products amalgamated over finite subgroups, and under HNN extensions 
with finite associated subgroups. 

We briefly recall the definitions of a graph of groups and its funda- 
mental group; a detailed introduction can be found in |38| . Let 1" be a 
finite, directed graph with (possibly) loops and multiple edges. We de- 
note by V{Y) and E{Y) the vertex and edge sets respectively of Y. Let 
a, uj : E{Y) V{Y) be the functions which take each edge to its start and 
end respectively. Suppose we have a fixed-point-free involution y ^ y on 
the edge set E(Y) which is orientation-reversing, that is, such that ya = yoj 
for all y £ E{Y). 

A graph of groups {G, Y) with underlying graph Y consists of 
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(i) for each vertex v € V{Y), a group G^; 

(ii) for each edge y G E{Y), a group Gy such that Gy = Gy] and 

(iii) for each edge y G E{Y), injective morphisms ay : Gy ^ Gya and 
ujy : Gy ^ Gyui such that ay = ujy for all y G E{Y). 

We assume that the groups G„ intersect only in the identity, and that they 
are disjoint from the edge set E{Y). For each v G V{Y)^ let {Xy \ Ry) be a 
monoid presentation for the vertex group G^, with the different generating 
sets Xy disjoint. Let B denote the (disjoint) union of E{Y) with all the sets 
X„. We define a group F{G,Y) by the monoid presentation 

F{G,Y) = {B \R, {v^V{Y)), 

yy = l {yeE{Y)), 

y{9^y)y = 90ty {y e E{Y),g £ Gy) ). 

Fix a vertex vq G V{Y). We say that a word w (z B* is of cycle type at vq if 
it is of the form w = ^^02/1^12/2^2 • • • 2/nWn where: 

(i) each yi G E{Y); 

(ii) yi • • • is a path in Y starting and ending at vq; 

(iii) Wo G X*^ ; 

(iv) ioi 1 < i < n, Wi £ X*^^. 

The images in F{G, Y) of the words of cycle type at vq form a subgroup 
7ri(G, Y, Vq) of F{G, Y), called the fundamental group of {G, Y) at vq. If the 
graph Y is connected then the fundamental group is (up to isomorphism) 
independent of the choice of vertex vq. The groups F{G, Y) and tti{G, Y, vq) 
are also easily seen to be independent of the presentations chosen for the 
vertex groups. 

Theorem 5.1. Let{G,Y) he a finite, connected, non-empty graph of finitely 
generated groups with finite edge groups. Then the fundamental group of 
{G, Y) has decidable rational subset problem if and only if every vertex group 
has decidable rational subset problem. Moreover, the equivalence is effective. 

Proof. Since each vertex group embeds into the fundamental group |HS|, one 
implication is immediate. 

For the converse, we use the notation defined above. Since the vertex 
groups are assumed to be finitely generated, we may assume that the gener- 
ating sets Xy are finite. Moreover, since the edge groups are finite and there 
are finitely many edges, we may assume without loss of generality that for 
every edge y, the sets Xya and Xy^^ contain a letter representing each non- 
identity element of Gyaay and Gy^^ojy, respectively. Let B be the (disjoint) 
union of all the sets X^ and the edges of Y . Then B \s a. finite generating 
set for the group F{G,Y). 

Fix a vertex vq G V{Y), and let P B* denote the set of words of cycle 
type at vq. We claim first that the intersection of P with the word problem 
W of F{G, Y) is RID. To show this we define an RID monadic rewriting 
system P over B with the following three types of rules: 

• If is a vertex and w G Xy represents the identity of Gy , then there 
is a rule w —>■ e; 
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• If y is an edge, then there is a rule yy — ^ e; 

• If y is an edge and h G Xya is a letter representing g G GyOy and 
w G X*^ represents gwy, then there is a rule ywy h. 

Since all the vertex groups are assumed to be RID, it follows easily 
from Theorem 13. II and Proposition 13.21 that T is an RID monadic rewriting 
system. Let L = {e}r~^. Since a singleton language is RID, we deduce by 
Corollary 14.21 that L is RID (as singletons are clearly RID languages). We 
claim that L is the intersection of the language P of words of cycle type at 
vq with the word problem W of F(G, Y). 

Clearly, the rewriting rules in T are relations satisfied in F(G,Y), from 
which it follows that L C W. It is also easy to see that the rewriting rules 
in r and their inverses preserve paths of cycle type at vq, so that L Q P. 
Thus, LCWnP. 

Conversely, suppose that w & W Ci P. Then we can write 

W = WoyiWiy2W2 ■ ■ ■ ynWn 

as per (i)-(iv) above. We proceed by induction on the parameter n (which 
we shall term the length of w) to show that w (z L. If n = 0, then w 
represents 1 in Gy^ and so w ^ e £ T, whence w ^ e, establishing w £ L. 

Assume that all elements of fl P of length at most k < n belong to L. 
Since w G WnP, it follows by |38| Theorem 1. 11] that there exists i such that 
yj+i = IJi and Wi represents an element gtOy^ for some g G Gy^ . There are two 
cases: either g = 1 ov g ^ 1. If g = 1, then we may apply the rule Wi ^ e fol- 
lowed by the rule yiyl — > e to show that if w' = w^yi . . . ■Wi-iWi+iyi+2 • • • Wn, 
then w w' . As w' £ W D P and has smaller length, it follows by induc- 
tion that w' G L, from which we obtain w £ L. If 17 7^ 1, let /i G Xy.a be 
a letter representing guy^. Then by definition yiWiYi h lies in F. Since 
yi+i = Yii if we set w' = w^yi . . . Wi-ihwi-\-iyiJ^2 ■ • • Wn, then w w' . Since 
w' £ ly nP represents the identity in F{G,Y) and its expression is shorter, 
we obtain by induction that w' =^>* e and so w e. Thus, we conclude that 
W n P = L and so is RID, as claimed. 

Now let X be a finite monoid generating set for the fundamental group 
■ki{G,Y,vq); this group is indeed finitely generated, namely by a set in cor- 
respondence with the disjoint union of the generating sets X^ and the set of 
edges not belonging to some spanning tree for Y |38,. Choose a morphism 
p : X* B* which takes each element x £ X to some word Wx £ P which 
represents the same element of tti{G, Y, vq) as x. Since P is a submonoid of 
B*, the image of p lies in P. Hence, for any word w £ X*, w lies in the word 
problem of tti{G,Y,vo) with respect to X if and only if wp lies in W, that 
is, if and only if wp lies mWr]P = L. So the word problem for tti{G, Y, vq) 
is an inverse morphic image of an RID language, and so by Proposition 13.21 
is RID. The theorem now follows from Theorem 13. II □ 

An HNN extension is the fundamental group of a graph of groups with a 
single vertex v and a matched pair y,y of loops at v. The vertex group G„ 
is the base group of the HNN construction, while the edge group Gy = Gy is 
isomorphic to the associated subgroups PHI- Hence, we obtain the following 
corollary. 
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Corollary 5.2. Let H he an HNN extension of a group G with finite as- 
sociated subgroups. Then H has decidable rational subset problem if and 
only if G has decidable rational subset problem. Moreover, the equivalence 
is effective. 

Similarly, an amalgamated free product corresponds to the fundamental 
group of a graph of groups with two vertices connected by a matched pair 
of edges |SH1) yielding the following result. 

Corollary 5.3. Let H he a free product of groups Gi and G2 amalgamated 
over a finite subgroup. Then H has decidable rational subset problem if and 
only if Gi and G2 have decidable rational subset problem. Moreover, the 
equivalence is effective. 

We remark that the proof of Theorem 15.11 establishes a more general 
fact. Recall that if C is a class of languages closed under inverse morphisms 
then the property of a finitely generated group having word problem in C is 
independent of the finite generating set chosen |20i Lemma 1]. 

Theorem 5.4. LetC be a class of languages closed under inverse morphism, 
left and right translation and closed under taking the ancestors of {e} for 
any C-monadic rewriting system. Then the fundamental group of a finite 
graph of finitely generated groups with finite edge groups has word problem 
in C if all the vertex groups have word problems in C. The converse is true 
if C is closed under intersection with regular languages. 

The assumptions of Theorem 15 . 41 are satisfied by, for example, the class of 
context-free languages (51 [7j. A well-known theorem of Muller and Schupp 
pS] augmented by a subsequent result of Dunwoody jT2] says that a group 
has context-free word problem if and only if it is virtually free. Our method 
therefore gives the following result, which can also be proved in many other 
ways, for instance from work of Karrass, Pietrowski and Solitar :28;. 

Corollary 5.5. Let (G,Y) be a finite, connected graph of groups with finite 
edge groups. Then the fundamental group of (G, Y) is virtually free if and 
only if every vertex group is virtually free. 

6. G-AUTOMATA AND RATIONAL SUBSETS 

In this section, we consider the rational subset problem in direct products. 
We show that if G is a finitely generated group and M a finitely generated 
monoid, then the rational subset problem for GxM is decidable exactly if the 
subsets of M defined by G- automata have uniformly decidable membership 
problem. This combines with a group-theoretic interpretation [25j of a well- 
known theorem of Chomsky and Schiitzenberger 9* to give a characterisation 
of rational subset membership in direct products of the form F x M with F 
a free group, in terms of the uniform decidability of membership for context- 
free subsets of M. 

Let G be a finitely generated group. Recall that a G-automaton over 
the alphabet S is a finite automaton P over G x S*. The G-automaton 
language accepted by P is the set of all words w € T,* such that {l,w) 
belongs to the rational subset recognised by P jl71 125j . The G-automaton 
languages are exactly the rational transductions of the word problem of 
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G |25L Proposition 2]. More generally, we say that a G-automaton over a 
monoid M is a finite automaton P over G x M. The G-automaton subset 
recognised by P is the set of all elements m ^ M such that (1, m) belongs to 
the rational subset recognised by P. It is easily seen that the G-automaton 
subsets are exactly the homomorphic images of G-automaton languages. 

Having fixed G and M and some finite generating sets A and B respec- 
tively, the G-automata subsets of M have a natural encoding as finite au- 
tomata over G x M. Thus, we may ask whether membership is uniformly 
decidable for G-automaton subsets of M; an argument similar to the proof 
of Proposition 12.11 shows that this property is independent of the choice 
of generating sets. The following result relates decidability properties of 
G-automaton subsets to the rational subset membership problem. 

Theorem 6.1. Let G he a finitely generated group, and M a finitely gener- 
ated monoid. Then the following are equivalent: 

(i) the rational subset problem for G x M is decidable; 

(ii) membership is uniformly decidable for G-automaton subsets of M . 

Proof. Let A and B be finite generating sets for G and M respectively, so 
that A\JB is a generating set for G x M. In view of our comments above, we 
may assume that all words and automata are encoded using these generating 
sets. 

Suppose first that (i) holds, that is, that the rational subset problem is 
decidable, and that we are given a finite automaton P over G x M and 
an element m ^ M. In view of our choice of generators, the word over B 
encoding m £ M also encodes the element {l,m) € G x M. Now m lies 
in the G-automaton language defined by P if and only if (l,m) lies in the 
rational subset defined by P. By assumption, this can be tested, so that (ii) 
holds 

Conversely, suppose (ii) holds, and that we are given a finite automaton 
P defining a rational subset of i? C G x M and an element (g, m) £ G x M. 
Since {g, m) is encoded as a word over Au B, we can easily compute a word 
representing ((7,1) and from that, a word representing {g~^,l). It follows 
that we can construct from P a finite automaton Q recognising the rational 
subset {g~^, 1)R. Now {g,m) lies in R if and only if (l,m) lies in {g~^, 1)R, 
that is, if and only if (1, m) is accepted by Q as a G-automaton. Once again, 
this can be tested, which shows that (i) holds and completes the proof. □ 

We note that, since property (i) in the statement of Theorem 16. II is sym- 
metric in G and M, we obtain the following corollary for G-automaton 
subsets of groups. 

Corollary 6.2. Let G and H he finitely generated groups. Then the follow- 
ing are equivalent: 

(i) membership is uniformly decidable for G-automata subsets of H; 

(ii) membership is uniformly decidable for H -automata subsets of G. 

We now turn our attention to the implications of Theorem l6.1l in the case 
that the group G is a finitely generated free group F of rank 2 or more. 
By |25l Theorem 7], which is essentially a group-theoretic restatement of 
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the Chomsky-Schutzenberger theorem 0111, the languages accepted by F- 
automaton are exactly the context-free languages, Combining with Theo- 
rem we immediately obtain the following corollary, where denotes a 
free group of rank n. 

Corollary 6.3. Let M he a finitely generated monoid. Then the following 
are equivalent: 

(i) the rational subset problem is decidable for the direct product F2 x M ; 

(ii) the rational subset problem is decidable for the direct product FnXM 
for all n > 0; 

(iii) membership is uniformly decidable for context-free subsets of M (en- 
coded as context-free grammars over a finite generating set). 

This leads to a new proof of the following result of Frougny, Sakarovitch 
and Schupp P^ . 

Corollary 6.4. The membership problem for context-free subsets of a free 
non-abelian group on at least 2 generators is undecidable. 

Proof. If F is a free non-abelian group then F x F has undecidable subgroup 
membership problem [30' '34' and hence undecidable rational subset problem. 
The result now follows from Corollary 16.31 □ 

Applying known results from language theory, we obtain the following. 

Theorem 6.5. Let F be a free group. The rational subset problem is decid- 
able for: 

(i) direct products F x A with A a finitely generated abelian group; 

(ii) direct products F x M with M a finitely generated commutative mon- 
oid; and 

(iii) direct products F x X* with X a finite set. 

Proof. In each case, by Corollary 16.31 it suffices to show that membership 
is uniformly decidable for context-free subsets of the monoid in question 
(encoded as context-free grammars over a finite generating set). This is 
well-known for the case of a free monoid X* j22| Section 6.3], so (iii) holds. 

Since abelian groups are examples of commutative monoids, it suffices 
now to prove case (ii). Let M be a commutative monoid generated by a 
finite subset X; then there is a surjective morphism p : X* M. Let 
a : X* — > be the canonical morphism from the free monoid X* to 
the free commutative monoid N'^ on X; clearly p factors through a via a 
morphism r : M. 

Now suppose we are given a context-free subset of M (encoded as a 
context-free grammar over X) and an element m & M encoded as a word 
w € X*. Let L be the language over X generated by the grammar. Then 
by Parikh's Theorem jHEj, we can effectively compute a regular language L' 
such that L'a = La. Hence Lp = L'p is a rational subset of M. But a 
result of Eilenberg and Schiitzenberger ^1] shows that the preimage of any 
rational subset of M under r is a rational subset of . Moreover, the proof 
is effective: one can effectively find a regular language L" C X* such that 
L"a = L'pT^^. So m G L if and only if wa £ L"a. Hence we have reduced 
our problem to the rational subset membership problem for N^. Now is 
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a finitely generated submonoid of 1?^ . It was observed by Grunschlag JU] 
that the description of rational subsets in commutative monoids as semilin- 
ear sets, due to Eilenberg and Schiitzenberger leads to an immediate 
solution of the rational subset membership problem for (and hence N'''") 
via integer programming. This completes the proof. □ 

7. Graph Groups 

In this section, we briefly discuss the subgroup membership problem and 
rational subset problem for the class of graph groups (which are also known 
as right-angled Artin groups, trace groups or free partially commutative 
groups). 

Let r be a finite undirected graph; we denote by l^(r) the vertex set of 
r, and by -E'(r) the edge set of F, which we view as a symmetric, refiexive 
subset of l^(r) X y(r). Recall that the graph group G(r) is the group with 
presentation 

(y(r) I e/ = /eforall(e,/)Gii;(r)). 

The extreme examples of graph groups, obtained when T has no edges 
or is complete, are free groups and free abelian groups respectively. In 
general, there are many properties of groups which are easily seen to hold 
for free groups and free abelian groups, but for radically different reasons. 
Establishing the extent to which such properties hold in general graph groups 
is often much more difficult, but can be very enlightening. Decidability of 
the subgroup membership problem and decidability of the rational subset 
problem are two such properties. 

A recent result of Kapovich, Weidmann and Myasnikov |22j shows that 
the subgroup membership problem is decidable for graph groups on finite 
graphs without chord-free cycles of length four or more. On the other hand, 
the graph group on a four-cycle is a direct product of non-abelian free groups; 
it follows from the results of Mikhailova |3fll I34j discussed above that the 
subgroup membership problem is undecidable for G(r) whenever T contains 
a chord-free four-cycle. Decidability of the subgroup membership problem 
seems to be open for graph groups G(r) where F contains chord-free cycles 
but not of length four; these groups do not contain a direct product of 
non-abelian groups as a subgroup ^26j, and so Mikhailova's result does not 
assist. The following question is an obvious starting point for research in 
this direction. 

Question 7.1. Is the subgroup membership problem decidable for the graph 
group on an n-cycle with n>5? 

Similarly, we have seen that the rational subset problem is decidable for 
free groups and free abelian groups. Combining Corollary 15.31 and Theo- 
rem ESI we immediately obtain decidability for a somewhat larger class. 

Corollary 7.2. The rational subset membership problem is decidable for 
free products of direct products of a free group with a free abelian group. 

In graph-theoretic terms, Corollarv 17.21 applies to G{T) where every con- 
nected component of T is the join (see of a complete graph and a graph 
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with no edges. On the other hand, wc saw above that the subgroup member- 
ship problem is undecidable for G(T) where F contains a four-cycle without 
chords, so the rational subset problem is also undecidable in these cases. 
Many cases remain open; these are summarised in the following question. 

Question 7.3. Is the rational subset membership problem decidable for G{T) 
when 

(i) r is a three-edge line? 

(ii) r has no chord- free cycles of length four or more? 

(iii) r is an n-cycle with n> 5? 

(iv) r contains no chord-free four-cycles? 
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